feat(storage): sync async with main branch#1744
Open
googlyrahman wants to merge 59 commits intogoogleapis:asyncfrom
Open
feat(storage): sync async with main branch#1744googlyrahman wants to merge 59 commits intogoogleapis:asyncfrom
googlyrahman wants to merge 59 commits intogoogleapis:asyncfrom
Conversation
…upload (googleapis#1654) feat: send entire object checksum in the final api call of resumable upload fixes b/461994245 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
feat: Support urllib3 >= 2.6.0 **Context**: * This library implements a custom decoders ( `_GzipDecoder` , `_BrotliDecoder` ) which inherit from `urllib3.response.ContentDecoder` * Interface of `urllib3.response.ContentDecoder` was changed in [2.6.0](https://urllib3.readthedocs.io/en/stable/changelog.html#id1) to fix security vulnerability for highly compressed data reads. (Decompression bombs) Hence we need to change our interfaces as well. **Changes** * Add `max_length` param on decompress method, provide default value of -1 (same as urllib3's decompress) * Provide backwards compatibility ( ie urllib3 <= 2.5.0)
🤖 I have created a release *beep* *boop* --- ## [3.7.0](googleapis/python-storage@v3.6.0...v3.7.0) (2025-12-09) ### Features * Auto enable mTLS when supported certificates are detected ([googleapis#1637](googleapis#1637)) ([4e91c54](googleapis@4e91c54)) * Send entire object checksum in the final api call of resumable upload ([googleapis#1654](googleapis#1654)) ([ddce7e5](googleapis@ddce7e5)) * Support urllib3 >= 2.6.0 ([googleapis#1658](googleapis#1658)) ([57405e9](googleapis@57405e9)) ### Bug Fixes * **bucket:** Move blob fails when the new blob name contains characters that need to be url encoded ([googleapis#1605](googleapis#1605)) ([ec470a2](googleapis@ec470a2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com>
chore: add gcs-fs as CODEOWNERS
fix: close write object stream always. otherwise Task will remain for long time until GC kills it and it'll throw this `"Task was destroyed but it is pending!"`
…oogleapis#1636) fix(experimental): no state lookup while opening bidi-write stream
This PR adds support for Python 3.14 to the library. Key changes include: - Updating `setup.py` to include the Python 3.14 classifier. - Updating `testing/constraints-3.14.txt` to specify `grpcio >= 1.75.1`. - Updating `noxfile.py` to include 3.14 sessions and set default to 3.14. - Updating `.github/sync-repo-settings.yaml` to include 3.14 in required checks. Tests were run locally using the `python-multi` Docker image for Python 3.9 and 3.14. Some unit tests failed due to environment issues within the container, but these are not expected to occur in the CI environment. --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
…nt (googleapis#1668) skip failing samples due to public access prevention enforcement. More Details on b/469643064
chore: add system test for cloud path
chore: add system 3.9 tests as required
feat: expose persisted size in MRD (MultiRangeReader)
fix: add system test for opening with read_handle
chore: Prevent OS Login key accumulation in e2e tests
feat: compute chunk wise checksum for bidi_writes and send it via BidiWriteObjectRequest As a part of this change, also did a small refactoring * Moved the precondition check to __utils.py_ file
chore: Add Sys test large obj
feat: make flush size configurable
chore: skip kms tests until b/470276398 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…nning zonal system tests (googleapis#1691) chore: optimization to reduce number of open TCP connections while running zonal system tests 1. Increase `ulimit -n 10000` before ssh'ing into the VM where system tests for zonal buckets are running. 2. Delete `mrd` and `writer` instance and trigger `gc.collect()` ( this alone should suffice but increasing doing the above optimization to avoid future issues.
feat: implement `append_from_file`
chore:delete topic after creation part of b/470069573
…is#1694) Revert "skip notification tests until b/470069573 is fixed"
- Add bidi stream retry manager. It will the main class that would be responsible for retries. - Integrate MRD with bidi stream retry manager and reads resumption strategy - Added conformance tests for all error scenarios and smart resumption for bidi reads - Refactored MRD's `download_ranges()` method, which will now use retry manager's `execute()` method to send the requests - Modified the logic to parse and handle the `BidiReadObjectRedirectedError`. - Modified cloud build configuration to also pull the latest changes of the current PR. - Formatted few more files using black formatter
…h generation (googleapis#1698) chore(tests): add test for reading unfinalized appendable objects with generation
Adding writes resumption strategy which will be used for error handling of bidi writes operation.
…#1699) Earlier the last chunk was being flushed while calling the close() method. Now it will be done inside the append method itself.
Bump the current version in `.librarian/state.yaml` to `3.7.0` to cater for googleapis#1621 which was released on December 9 2025. This will address the issue in googleapis#1701 where librarian sets the version to `3.7.0` which has already been released. The reason that this manual PR is needed is because the release on December 9th used a different tool `release-please` which does not update the `.librarian/state.yaml` file.
) PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v0.7.0 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91 <details><summary>google-cloud-storage: 3.8.0</summary> ## [3.8.0](googleapis/python-storage@v3.7.0...v3.8.0) (2026-01-13) ### Features * expose persisted size in mrd (googleapis#1671) ([0e2961b](googleapis@0e2961be)) * implement &googleapis#34;append_from_file&googleapis#34; (googleapis#1686) ([1333c95](googleapis@1333c956)) * compute chunk wise checksum for bidi_writes (googleapis#1675) ([139390c](googleapis@139390cb)) * flush the last chunk in append method (googleapis#1699) ([89bfe7a](googleapis@89bfe7a5)) * add write resumption strategy (googleapis#1663) ([a57ea0e](googleapis@a57ea0ec)) * add bidi stream retry manager. (googleapis#1632) ([d90f0ee](googleapis@d90f0ee0)) * make flush size configurable (googleapis#1677) ([f7095fa](googleapis@f7095faf)) ### Bug Fixes * no state lookup while opening bidi-write stream ([2d5a7b1](googleapis@2d5a7b16)) * no state lookup while opening bidi-write stream (googleapis#1636) ([2d5a7b1](googleapis@2d5a7b16)) * close write object stream always (googleapis#1661) ([4a609a4](googleapis@4a609a4b)) * add system test for opening with read_handle (googleapis#1672) ([6dc711d](googleapis@6dc711da)) </details> --------- Co-authored-by: Victor Chudnovsky <vchudnov@google.com>
…quests in async streams. Gracefully close streams. (googleapis#1700) fix(experimental): implement requests_done method to signal end of requests in async streams. Gracefully close streams.
…ng existing objects and add `is_stream_open` support (googleapis#1709) feat(ZonalBuckets): add support for `generation=0` to prevent overwriting existing objects feat(ZonalBuckets): add `is_stream_open` property to AsyncAppendableObjectWriter for stream status check --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…oogleapis#1705) feat(samples): add samples for appendable objects writes and reads --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Adding a support to pass `generation` in the constructor of MRD. It is done to make it compatible with the other public interfaces. Earlier, `generation_number` was exposed as the parameter which would still be valid after these changes. But using `generation` is more preferred.
Expose `DELETE_OBJECT` in `AsyncGrpcClient`
When `writer.close()` is called without setting finalize_on_close flag, we need to get two responses: 1) to get the persisted_size 2) eof response That's why added a check if the first response is not eof, then again receive the response from the stream.
chore: Add README for running zonal buckets samples
This method can be used to fetch the metadata of an object using the async grpc API.
) PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v1.0.2-0.20251119154421-36c3e21ad3ac Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91 <details><summary>google-cloud-storage: 3.9.0</summary> ## [3.9.0](googleapis/python-storage@v3.8.0...v3.9.0) (2026-02-02) ### Features * update generation for MRD (googleapis#1730) ([08bc708](googleapis@08bc7082)) * add get_object method for async grpc client (googleapis#1735) ([0e5ec29](googleapis@0e5ec29b)) * Add micro-benchmarks for reads comparing standard (regional) vs rapid (zonal) buckets. (googleapis#1697) ([1917649](googleapis@1917649f)) * Add support for opening via `write_handle` and fix `write_handle` type (googleapis#1715) ([2bc15fa](googleapis@2bc15fa5)) * add samples for appendable objects writes and reads ([2e1a1eb](googleapis@2e1a1eb5)) * add samples for appendable objects writes and reads (googleapis#1705) ([2e1a1eb](googleapis@2e1a1eb5)) * add context manager to mrd (googleapis#1724) ([5ac2808](googleapis@5ac2808a)) * Move Zonal Buckets features of `_experimental` (googleapis#1728) ([74c9ecc](googleapis@74c9ecc5)) * add default user agent for grpc (googleapis#1726) ([7b31946](googleapis@7b319469)) * expose finalized_time in blob.py applicable for GET_OBJECT in ZB (googleapis#1719) ([8e21a7f](googleapis@8e21a7fe)) * expose `DELETE_OBJECT` in `AsyncGrpcClient` (googleapis#1718) ([c8dd7a0](googleapis@c8dd7a0b)) * send `user_agent` to grpc channel (googleapis#1712) ([cdb2486](googleapis@cdb2486b)) * integrate writes strategy and appendable object writer (googleapis#1695) ([dbd162b](googleapis@dbd162b3)) * Add micro-benchmarks for writes comparing standard (regional) vs rapid (zonal) buckets. (googleapis#1707) ([dbe9d8b](googleapis@dbe9d8b8)) * add support for `generation=0` to avoid overwriting existing objects and add `is_stream_open` support (googleapis#1709) ([ea0f5bf](googleapis@ea0f5bf8)) * add support for `generation=0` to prevent overwriting existing objects ([ea0f5bf](googleapis@ea0f5bf8)) * add `is_stream_open` property to AsyncAppendableObjectWriter for stream status check ([ea0f5bf](googleapis@ea0f5bf8)) ### Bug Fixes * receive eof while closing reads stream (googleapis#1733) ([2ef6339](googleapis@2ef63396)) * update write handle on every recv() (googleapis#1716) ([5d9fafe](googleapis@5d9fafe1)) * implement requests_done method to signal end of requests in async streams. Gracefully close streams. (googleapis#1700) ([6c16079](googleapis@6c160794)) * implement requests_done method to signal end of requests in async streams. Gracefully close streams. ([6c16079](googleapis@6c160794)) * instance grpc client once per process in benchmarks (googleapis#1725) ([721ea2d](googleapis@721ea2dd)) * Fix formatting in setup.py dependencies list (googleapis#1713) ([cc4831d](googleapis@cc4831d7)) * Change contructors of MRD and AAOW AsyncGrpcClient.grpc_client to AsyncGrpcClient (googleapis#1727) ([e730bf5](googleapis@e730bf50)) </details>
Automated: Migrate {target_path} from gsutil to gcloud storage
This CL is part of the on going effort to migrate from the legacy
`gsutil` tool to the new and improved `gcloud storage` command-line
interface.
`gcloud storage` is the recommended and modern tool for interacting with
Google Cloud Storage, offering better performance, unified
authentication, and a more consistent command structure with other
`gcloud` components. 🚀
### Automation Details
This change was **generated automatically** by an agent that targets
users of `gsutil`.
The transformations applied are based on the [gsutil to gcloud storage
migration guide](http://go/gsutil-gcloud-storage-migration-guide).
### ⚠️ Action Required: Please Review and Test Carefully
While we have based the automation on the migration guide, every use
case is unique.
**It is crucial that you thoroughly test these changes in environments
appropriate to your use-case before merging.**
Be aware of potential differences between `gsutil` and `gcloud storage`
that could impact your workflows.
For instance, the structure of command output may have changed,
requiring updates to any scripts that parse it. Similarly, command
behavior can differ subtly; the `gcloud storage rsync` command has a
different file deletion logic than `gsutil rsync`, which could lead to
unintended file deletions.
Our migration guides can help guide you through a list of mappings and
some notable differences between the two tools.
Standard presubmit tests are run as part of this CL's workflow. **If you
need to target an additional test workflow or require assistance with
testing, please let us know.**
Please verify that all your Cloud Storage operations continue to work as
expected to avoid any potential disruptions in production.
### Support and Collaboration
The `GCS CLI` team is here to help! If you encounter any issues, have a
complex use case that this automated change doesn't cover, or face any
other blockers, please don't hesitate to reach out.
We are happy to work with you to test and adjust these changes as
needed.
**Contact:** `gcs-cli-hyd@google.com`
We appreciate your partnership in this important migration effort!
#gsutil-migration
Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com>
Add a warning that generation_number will be deprecated in the next major release.
The PR sync the main and async branch. --------- Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Adds async credential wrapper for async client. Context: - The standard google auth credentials are currently synchronous, and it's asynchronous credentials classes are either not available or marked private. - Credential retrieval and refreshing will remain synchronous under the hood. Rationale: As authentication tokens typically possess an expiration lifetime of one hour, the blocking time required for token fetching occurs infrequently. The performance impact of blocking (or utilizing a separate thread for offloading) once per hour is deemed negligible when weighed against the considerable engineering cost of developing and maintaining a asynchronous authentication.
…leapis#1655) This PR introduces an abstract class to support the upcoming async client for the Python SDK. This refactor defines the public interface and enables code sharing between sync and async client, without introducing new logic.
…oogleapis#1696) 1. Add async client implementation. 2. Add AsyncHTTPIterator deriving from google.api_core.page_iterator_async.AsyncIterator as an alternative to google.api_core.page_iterator.HTTPIterator The AsyncHTTPIterator doesn't exists, and hence needs to be implemented.
product-auto-label
bot
added
size: xl
googlyrahman
changed the title
googlyrahman
changed the title
Contributor
Summary of ChangesHello @googlyrahman, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates a substantial set of enhancements and new features for the asynchronous gRPC client within the Google Cloud Storage library. The core focus is on improving the reliability and functionality of interactions with Zonal Buckets and appendable objects, moving previously experimental APIs to a stable state. It also lays the groundwork for robust performance evaluation through new benchmarking tools. Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request represents a significant synchronization of the async implementation with the main branch, promoting many experimental features to stable. Key changes include moving async modules out of the _experimental namespace, adding get_object and delete_object to the AsyncGrpcClient, and introducing comprehensive retry and resumption strategies for bidi streams. The PR also includes a substantial number of new system tests, conformance tests, and performance microbenchmarks to validate these features. I've identified a few minor issues in the changelog and a docstring that could be improved for clarity.
| * Add micro-benchmarks for reads comparing standard (regional) vs rapid (zonal) buckets. (#1697) ([1917649fac41481da1adea6c2a9f4ab1298a34c4](https://github.com/googleapis/python-storage/commit/1917649fac41481da1adea6c2a9f4ab1298a34c4)) | ||
| * send `user_agent` to grpc channel (#1712) ([cdb2486bb051dcbfbffc2510aff6aacede5e54d3](https://github.com/googleapis/python-storage/commit/cdb2486bb051dcbfbffc2510aff6aacede5e54d3)) | ||
| * add samples for appendable objects writes and reads (#1705) ([2e1a1eb5cbe1c909f1f892a0cc74fe63c8ef36ff](https://github.com/googleapis/python-storage/commit/2e1a1eb5cbe1c909f1f892a0cc74fe63c8ef36ff)) | ||
| * add samples for appendable objects writes and reads ([2e1a1eb5cbe1c909f1f892a0cc74fe63c8ef36ff](https://github.com/googleapis/python-storage/commit/2e1a1eb5cbe1c909f1f892a0cc74fe63c8ef36ff)) |
Contributor
There was a problem hiding this comment.
This changelog entry is a duplicate of the one on the previous line. Please remove it to avoid redundancy.
Comment on lines
+27
to
+28
| * add support for `generation=0` to prevent overwriting existing objects ([ea0f5bf8316f4bfcff2728d9d1baa68dde6ebdae](https://github.com/googleapis/python-storage/commit/ea0f5bf8316f4bfcff2728d9d1baa68dde6ebdae)) | ||
| * add `is_stream_open` property to AsyncAppendableObjectWriter for stream status check ([ea0f5bf8316f4bfcff2728d9d1baa68dde6ebdae](https://github.com/googleapis/python-storage/commit/ea0f5bf8316f4bfcff2728d9d1baa68dde6ebdae)) |
Contributor
There was a problem hiding this comment.
These two entries are redundant as they are already covered by the more comprehensive entry on line 26. Please remove them to improve clarity.
| * update write handle on every recv() (#1716) ([5d9fafe1466b5ccb1db4a814967a5cc8465148a2](https://github.com/googleapis/python-storage/commit/5d9fafe1466b5ccb1db4a814967a5cc8465148a2)) | ||
| * Fix formatting in setup.py dependencies list (#1713) ([cc4831d7e253b265b0b96e08b5479f4c759be442](https://github.com/googleapis/python-storage/commit/cc4831d7e253b265b0b96e08b5479f4c759be442)) | ||
| * implement requests_done method to signal end of requests in async streams. Gracefully close streams. (#1700) ([6c160794afded5e8f4179399f1fe5248e32bf707](https://github.com/googleapis/python-storage/commit/6c160794afded5e8f4179399f1fe5248e32bf707)) | ||
| * implement requests_done method to signal end of requests in async streams. Gracefully close streams. ([6c160794afded5e8f4179399f1fe5248e32bf707](https://github.com/googleapis/python-storage/commit/6c160794afded5e8f4179399f1fe5248e32bf707)) |
Contributor
There was a problem hiding this comment.
This changelog entry is a duplicate of the one on the previous line. Please remove it.
Comment on lines
+5039
to
+5045
| """If this object has been soft-deleted, returns the time at which it will be permanently deleted. | ||
|
|
||
| :rtype: :class:`datetime.datetime` or ``NoneType`` | ||
| :returns: | ||
| (readonly) The time that the object will be permanently deleted. | ||
| Note this property is only set for soft-deleted objects. | ||
| """ |
Contributor
There was a problem hiding this comment.
The docstring for the finalized_time property appears to be incorrect. It states that this property is for soft-deleted objects, which is confusingly similar to the purpose of hard_delete_time. The finalizedTime field in the API typically refers to when an object's generation was fixed (e.g., for appendable objects). Please update the docstring to accurately describe this property to avoid confusion for users.
"""The time at which the object became finalized and its generation was fixed.
See: https://cloud.google.com/storage/docs/json_api/v1/objects#finalizedTime
:rtype: :class:`datetime.datetime` or ``NoneType``
:returns:
(readonly) The time that the object was finalized.
Note this property is only set for objects that have been finalized.
"""
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync async with main branch